feat(store): iaf migrate-store + dual-store contract suite (epic #540 phase 3d)#547
Merged
Merged
Conversation
…phase 3d) Closes the open Phase 3 deliverables that turn the new store abstraction into something users can actually move data through: - iaf migrate-store --from <kind> --src <path> --to <kind> --dst <path> delegates to dst.copy_from(src), so it is incremental, restartable, and tier-aware: when the destination is a local-tiered store, identical OHLCV chunks are written exactly once across the destination regardless of how many bundles reference them (Phase 3c invariant). Optional --handles subset selector for partial migrations. - migrate_store() programmatic helper for in-process pipelines. - BacktestStoreContractTest: a parameterised conformance suite that runs identical scenarios against every concrete store implementation (LocalDirStore, LocalTieredStore today, future remote stores tomorrow). Catches divergence as a failing subTest with the store class name in the label. Covers Protocol + SupportsCopyFrom conformance, write/open round-trip, summary_only, exists, idempotent delete, missing-handle errors, listing, iter_index_rows, and copy_from with both full and subset handle selection. - bug fix in LazyOhlcvDict: items() and values() were inheriting the empty backing dict's iteration, so any code path that did 'for k, v in bt.ohlcv.items()' silently dropped every blob after a tiered round-trip. Now both methods walk the manifest and materialise lazily on access. Caught by the migration dedup test. Note on what is *not* in this PR: byte-identical Tier-2 -> Backtest reassembly (so .iafbt could become export-only) is intentionally deferred. The current model where the bundle is canonical and Tier-1/2/3 are derived is simpler, preserves the existing round-trip contract bit-for-bit, and is what every test in the contract suite already exercises against both stores. Targeted suite (backtest_store + backtest_index + cli): 128 / 128 passing + 26 subTests. Full non-scenario suite: 1705 / 1705 passing with no regressions from the LazyOhlcvDict fix.
…L dashboard Two new sections in examples/storage_layer_demo/demo.py: - 6b. _print_backtest_full_report(): per-run breakdown (window / days / orders / trades / positions / final_value), end-of-backtest positions snapshot, first few trades, and a richer slice of per-run BacktestMetrics (cagr, annual_volatility, max_drawdown_absolute, gross_profit/loss, best_trade, max consecutive wins/losses) with safe n/a fallbacks. Built on top of the existing compact _print_backtest_report(). - 9. Storage layer -> HTML dashboard: wires the Tier-1 SQLite index, the Tier-2 LocalDirStore and the BacktestReport HTML dashboard end-to-end. rank_index() picks the top-N bundles from SQLite alone, store.open(handle) materialises just those via the BacktestStore protocol, BacktestReport(backtests=[...]).save() renders a self-contained interactive HTML dashboard. Demonstrates that the new storage layer plugs straight into the existing reporting stack with no glue code. README updated to describe both new sections.
- New feature bullet linking the storage_layer_demo - New '<details>' section explaining Tier-1 SQLite index, Tier-2 BacktestStore adapters (LocalDirStore / LocalTieredStore) and Tier-3 content-addressed OHLCV chunks - Python + CLI workflow showing the canonical pattern: build_index -> rank_index -> store.open(handle) -> BacktestReport(backtests=[...]).save(...) - Links to examples/storage_layer_demo/ for the runnable end-to-end
Add a 'From backtest results to a report' subsection under 'Backtest Analysis & Dashboard' demonstrating the canonical paths from a Backtest (or list of Backtests) to a BacktestReport: - single event-driven app.run_backtest(...) - a sweep via app.run_vector_backtests(..., backtest_storage_directory=...) - loading a persisted folder back via BacktestReport.open(directory_path=..., workers=-1) Cross-links to the Backtest Storage Layer section for sweeps that scale into the thousands.
New 'Getting Started/Backtest Storage Layer' page covering:
- mental model (Tier-1 SQLite / Tier-2 Parquet / canonical .iafbt /
Tier-3 content-addressed OHLCV)
- the BacktestStore protocol and when to pick LocalDirStore vs
LocalTieredStore
- the canonical 5-step developer workflow:
run sweep -> build index -> filter/rank in SQLite ->
materialise winners -> render report
- 'Avoid overloading your report.html': size-vs-bundle table,
the BacktestReport.open(directory_path=...) anti-pattern,
rules of thumb for narrow vs mega-reports
- pointers to examples/storage_layer_demo and the migrate-store CLI
Wired into the Getting Started sidebar between backtest-reports and
deployment, and added a tip block on backtest-reports.md pointing to
it for users with thousands of backtests.
a0751e9 to
adfeaa0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 3d —
iaf migrate-store+ dual-store contract suiteStacked on #546 (Phase 3c). Closes the open Phase 3 deliverables of epic #540.
What's in the box
1.
iaf migrate-storeCLIDelegates to
dst.copy_from(src), which means the migration is:local-tieredstore, identical OHLCV chunks are written exactly once across the destination regardless of how many bundles reference them (the Phase 3c invariant).--handles a,b,cfor partial migrations.A
migrate_store(...)programmatic helper is also exposed for in-process pipelines.2.
BacktestStoreContractTest— parameterised dual-store suiteA single
unittest.TestCaseruns an identical scenario against every concreteBacktestStoreimplementation (LocalDirStore,LocalTieredStore) usingsubTest(store=label). Adding a future store (e.g. a remoteS3BacktestStore) is one entry in_STORES = [...]away from full conformance coverage.Coverage:
SupportsCopyFromruntime conformance.write→openround-trip preservesalgorithm_id.summary_onlyhonoured.existsreflects writes; idempotentdeleteremoves the handle.StoreHandleNotFoundError.iter_handlesand__len__agree with the written set.iter_index_rowsyields one row per bundle.copy_fromworks for both full migration and a handle subset.3. Bug fix:
LazyOhlcvDict.items()/.values()The previous implementation inherited the empty backing dict's iteration, so any code path doing
silently dropped every blob after a tiered round-trip — including the migration code we're adding here. Caught by
test_ohlcv_dedup_during_migrationand fixed by walking the manifest with lazy materialisation, matching the existing__iter__/__getitem__semantics.What is not in this PR
Byte-identical Tier-2 →
Backtestreassembly (so.iafbtcould become export-only) is intentionally deferred. The current model where the bundle is canonical and Tier-1/2/3 are derived is simpler, preserves the existing round-trip contract bit-for-bit, and is what every test in the contract suite already exercises against both stores. A future slice can promote Tier-2 to authoritative once we have a stronger reassembly story for the envelope's metadata fields.Tests
tests/cli/test_migrate_store_command.py— 7 tests (programmatic API + CLI: round-trip, handle subset, OHLCV dedup across migration, unknown-kind validation).tests/services/backtest_store/test_store_contract.py— 8 contract tests × 2 stores = 16 subTests.tests/services/backtest_store/+tests/services/backtest_index/+tests/cli/): 128 / 128 passing + 26 subTests.tests/minustests/scenarios): 1705 / 1705 passing with no regressions from theLazyOhlcvDictfix.Stack
Phase 3 of epic #540 is now feature-complete.